
                  QLogic 57710/57711/57712/578xx 10/20 GigE Driver
                                  For Solaris

                  Copyright (c) 2014 QLogic Corporation
                              All rights reserved.

Contents

I.    Introduction

II.   Supported Devices

III.  Installing the Driver

IV.   Uninstalling Driver

V.    Upgrading the Driver

VI.   Configuring the Driver

VII.  Memory Usage

VIII. Interrupt Management

IX.   Tuning

X.    Debugging

XI.   FCoE


I. Introduction
   ============

This file describes how to install and use the Solaris driver for QLogic's
577xx/578xx 10/100/1000/2500/10000/20000 Mbps Ethernet Network Interface
Controllers.  Please refer to the 'bnxe' manual page and 'bnxe.conf'
configuration file for details on how to configure the driver.

The driver is released in two formats (<os> = Solaris version, <arch> = system
architecture, X.Y.Z = version):

* Release
  1. BRCMbnxe-<platform>-<arch>-X.Y.Z.pkg   : package datastream format
  2. BRCMbnxe-<platform>-<arch>-X.Y.Z.tar.Z : compressed TAR format

* Debug
  1. BRCMbnxe-<platform>-<arch>-X.Y.Z-dbg.pkg   : package datastream format
  2. BRCMbnxe-<platform>-<arch>-X.Y.Z-dbg.tar.Z : compressed TAR format

Packages are built for each Solaris release supported.  Oracle does NOT
guarantee kernel driver binary compatibility between Solaris update releases.

Note that a DU installation image is not distributed by QLogic.  Solaris DU
images can be used to install the driver into the system both during system
installation and/or after the system has been installed and booted.  These
images can be easily created for various physical media type from either of the
distributed BRCMbnxe formats.  See the Solaris Installation guide for more
detailed information.

This driver only works with the GLDv3 layer interface.


II. Supported Devices
    =================

The following families of the QLogic 577xx/578xx 10/20Gb Ethernet Controller
are supported by this driver:

  * BCM57710
  * BCM57711
  * BCM57712
  * BCM578xx


III. Installing the Driver
     =====================

III(a). Driver update during Solaris Installation

The driver update (DU) images are used to install the driver during the
installation of Solaris. Using the Solaris text installer there is an
option to "Apply driver updates". Selecting this will bring up another
menu asking for the location of the media. The two easiest options are
either a CD/DVD or a USB Flash drive.

For a CD/DVD simply burn the BRCMbge-<platform>-<arch>-X.Y.Z_DU.iso image
to a disk. Then when prompted for the media location, eject the Solaris disk
and insert the driver update disk. Select the CD/DVD location option and the
driver will be installed. See below for possible errors. Once complete eject
the driver update disk and insert the Solaris disk.

For a USB Flash drive a "DU" directory must be located at the top level of
the drive. Note that the DU tar contains some symlinks which do not transfer
over properly to a FAT32 formatted flash drive. Follow these steps:

1. Mount flash drive:
   # mount /dev/<usb_flash> /media/usb

2. Untar DU image and copy to the flash drive:
   # tar -xvf BRCMbge-<platform>-<arch>-X.Y.Z_DU.tar
   # cp -r BRCMbge-<platform>-<arch>-X.Y.Z_DU/DU /media/usb

3. Fix the symlinks on the flash media. Take note of the original links
   you see after untarring the image. The links are based on the CPU
   architecture and differ between SPARC and x86. For example with x86:
   # cd /media/usb/DU/sol_210
   # cp -r i386 i86pc

4. Unmount flash drive:
   # umount /media/usb

Now simply insert the flash drive into the Solaris system and select the mass
storage device option. See below for possible errors. Once complete pull out
the flash drive.

If the driver does not install properly take note of any warning or error
messages presented. You'll have the option to drop into a shell and see the
actual log output from the driver update installation. Chances are there might
be a conflict with a pre-existing package or the inbox version of the driver.
This can be fixed by running "pkgrm <pkg>" or "rem_drv <drv>" appropriately.

III(b). Post Solaris Installation

1. Change directory to where BRCMbnxe-<platform>-<arch>-X.Y.Z.pkg resides.
2. pkgadd -d BRCMbnxe-<platform>-<arch>-X.Y.Z.pkg

   OR

1. Change directory to where BRCMbnxe-<platform>-<arch>-X.Y.Z.tar.Z resides.
2. uncompress BRCMbnxe-<platform>-<arch>-X.Y.Z.tar.Z
   tar -xvf BRCMbnxe-<platform>-<arch>-X.Y.Z.tar
   pkgadd -d .

2(!). If installing on a fresh install of Solaris you will have a conflict
   with the inbox driver which must first be removed. The "check_install"
   script in the BRCMbnxe package will prevent installation over an existing
   driver so examine the output from pkgadd carefully. The inbox driver
   cannot be uninstalled with the pkgrm command since there are no package
   remove scripts in the package directory under /var/sadm/pkg. Use the
   following steps to remove the inbox driver before performing a pkgadd of
   BRCMbnxe:

   a. Remove the package directory:
        rm -rf /var/sadm/pkg/BRCMbnxe
        [or]
        mv /var/sadm/pkg/BRCMbnxe /var/sadm/pkg/BRCMbnxe.inbox

   b. Remove the drivers:
        rem_drv bnxef
        rem_drv bnxe

   c. Verify bnxe and bnxef is not running:
        modinfo | grep bnxe

   d. If they are running then try to unload them:
        modunload -i `modinfo | grep bnxef | awk '{ print $1; }'`
        modunload -i `modinfo | grep bnxe | awk '{ print $1; }'`

   e. If they don't unload then reboot.

   f. Now you can perform the pkgadd as normal in step 2 above.

3. Determine the bnxe instance numbers (#)
     Solaris 10: dladm show-dev
     Solaris 11: dladm show-phys

4. Static IP:
     Solaris 10: ifconfig bnxe[#] plumb set <ip>/<cidr> up
     Solaris 11: See netcfg/netadm documentation.

4. DHCP:
     Solaris 10: ifconfig bnxe[#] plumb up
                 ifconfig bnxe[#] dhcp
     Solaris 11: See netcfg/netadm documentation.

To make these changes permanent, follow these steps:

Solaris 10:

  1. Use your favorite text editor (e.g. vi) and create a file named
     hostname.bnxe[#] in the /etc directory.  For static IP add the <ip>/<cidr>
     information for the interface to this file.  For DHCP leave the file empty
     and create another empty file named dhcp.bnxe[#] in /etc.

  2. Reboot.

Solaris 11: See netcfg/netadm documentation.

Please view the Solaris Networking Administration guide for more detailed
information.


IV. Uninstalling Driver
    ===================

1. For each bnxe interface that is plumbed:
     Solaris 10: ifconfig bnxe[#] unplumb
     Solaris 11: See netcfg/netadm documentation.

2. pkgrm BRCMbnxe


V. Upgrading the Driver
   ====================

To upgrade the QLogic driver package to the current version, you must first
uninstall the previous driver version from the system.  Once the previous
driver has been removed, you can follow any of the installation methods in this
document to install the new driver version.

Please note that QLogic does not support installing multiple instances of
the driver on a single system.  Only a single version of the bnxe driver can be
loaded at any one time.


VI. Configuring the Driver
    ======================

The bnxe driver can be configured via the bnxe.conf file installed under
/kernel/drv.  There are many configuration items available and all are
thoroughly documented in bnxe.conf.  When this config file is modified the
system must be either rebooted or the driver unloaded and reconfigured using
the 'update_drv' admin command.

All configurations can be specified per-instance.  The format used is as
follows and each line must end with a semicolon:

  bnxe<#>_<config_item>=X;

So for "adv_autoneg_cap" you would use the following:

  bnxe0_adv_autoneg_cap=1;
  bnxe1_adv_autoneg_cap=0;
  bnxe2_adv_autoneg_cap=1;
  bnxe3_adv_autoneg_cap=1;

If a configuration item is not specified for a specific instance then the
default value will be used.  The default value used by all instances can be
overridden using:

  default_<config_item>=X;

For boolean values 1 = TRUE and 0 = FALSE.


VII. Memory Usage
     ============

The number of Rx/Tx buffer descriptors specified in the configuration file can
have a detrimental affect on memory usage.  If the counts are too high, DMA
allocations can fail thereby affecting other drivers loaded on the system.  If
DMA allocations fail during system initialization and/or boot then there is a
chance the system won't boot.  This behavior is an implementation constraint of
the Solaris OS.  Additionally, it has been seen that the amount of DMA
allocation space available on a system running in 32-bit mode is less than when
running as 64-bit.

For a *single* Rx descriptor the following is allocated:

  . 1 DMA handle
  . 1 DMA memory buffer that is 'mtu' in size
  . approx 1K memory overhead

For a *single* Tx descriptor the following is allocated:

  . 9 DMA handles for sending chained mblks
  . 1 DMA double copy memory buffer that is 'mtu' in size
  . approx 1K memory overhead

Note that the number of DMA handles available in the system scales with the
amount of RAM.  So with more RAM the descriptor counts can safely be increased.

The default number of Rx/Tx buffer descriptors is 1024 per ring.  Again, that
is per Rx and per Tx ring!  When using an adapter in multi-function mode, where
the physical interface is partitioned into multiple virtual interfaces, the
number of configured descriptors is divided by four(4) ending up at 256 per
ring.  This is to keep the number of DMA allocations at a minimum.  After
installation it is suggested these descriptor counts be increased until
stability is guaranteed and the desired performance is reached.

For example, using the default setting of 1024 for the number of Rx and Tx
descriptors and the default setting of 4 rings, the approximate amount of
memory a single interface would consume is (all number approximate):

  Single Function Mode
  --------------------

  . Per Ring:
      Rx    -  1024 DMA handles, 2.5M (mtu=1500) / 10.5M (mtu=9216) of memory
      Tx    -  9216 DMA handles, 2.5M (mtu=1500) / 10.5M (mtu=9216) of memory
      Total - 10240 DMA handles, 5.0M (mtu=1500) / 21.0M (mtu=9216) of memory

  . Total for all rings (4):
      40960 DMA handles, 20M (mtu=1500) / 84M (mtu=9216) of memory

  Multi-Function Mode (#descs / 4)
  --------------------------------

  . Per Ring:
      Rx    -  256 DMA handles, 650k (mtu=1500) / 2.6M (mtu=9216) of memory
      Tx    - 2304 DMA handles, 650k (mtu=1500) / 2.6M (mtu=9216) of memory
      Total - 2560 DMA handles, 1.3M (mtu=1500) / 5.2M (mtu=9216) of memory

  . Total for all rings (4):
      10240 DMA handles, 5M (mtu=1500) / 21M (mtu=9216) of memory

  . Total for all virtual interfaces on physical port (4):
      40960 DMA handles, 20M (mtu=1500) / 84M (mtu=9216) of memory


VII. Interrupt Management
     ====================

The bnxe driver will allocate as many rings as specified via the "num_rings"
option in bnxe.conf.  Rx/Tx rings are only available when MSIX is used and each
ring will allocate one MSIX interrupt.  Additionally, one more interrupt is
needed for default processing.  For example if 4 rings are specified then a
total of 5 MSIX interrupts must be available to the driver.  If FCoE is enabled
then one additional interrupt would also be allocated for a total of 6 MSIX.
When bnxe cannot allocate all the needed MSIX interrupts then it will revert to
Fixed level legacy interrupts and a degraded performance mode.

By default, on x86, Solaris only allows 2 MSIX interrupts per driver instance.
This limit can be increased by editing the /etc/system file and adding the
following lines (reboot is required).  Set ddi_msix_alloc_limit accordingly:

  set ddi_msix_alloc_limit=8
  set pcplusmp:apic_intr_policy=1

If you have a system with many interfaces it is possible to reach the
allocation limit of MSIX interrupts.  Solaris allows a maximum of 32 MSIX
interrupts per "interrupt priority level".  Assume a system with four 57711
ports each running in multi-function mode.  Solaris will enumerate 16 bnxe
interfaces.  The number of rings must be set to one to ensure only 2 MSIX
interrupts are allocated per interface resulting in a total of 32 MSIX
interrupts allocated across all bnxe interfaces.  Unfortunately if another
interface was on or added to the system, interrupt allocations would start to
fail.

To ensure all interfaces are able to allocate their needed MSIX interrupts, the
workaround is to change the priority levels of specific interfaces.  Network
drivers are automatically assigned an interrupt priority level of 6 so changing
an interface's priority level to 5 is common.

0. First read the driver.conf man page for a background primer.

1. Find out the driver instance paths assigned on your system.

   % grep bnxe /etc/path_to_inst
   "/pci@0,0/pci8086,2779@1/pci14e4,1650@0" 0 "bnxe"
   "/pci@0,0/pci8086,2779@1/pci14e4,1650@0,1" 1 "bnxe"

2. Normally, the name of the driver is the last portion of the path but you
   should use the most appropriate PCI ID found in /etc/driver_aliases.
   Depending on how the hardware is layered there are cases where the name
   identified in path_to_inst won't work.  To figure out which name to use
   examine the output from 'prtconf -v' and match against the IDs specified
   in the driver_aliases file.

   % grep bnxe /etc/driver_aliases
   bnxe "pci14e4,164e"
   bnxe "pci14e4,164f"
   bnxe "pci14e4,1650"
   bnxe "pciex14e4,164e"
   bnxe "pciex14e4,164f"
   bnxe "pciex14e4,1650"

3. The parent of the driver is the entire path leading up to the name.

4. The unit-address is located after the final '@' in the path.

5. For example, changing both of the bnxe interfaces found the above
   path_to_inst to interrupt priority 5 we would use the following config lines
   to bnxe.conf:

   name = "pciex14e4,1650" parent = "/pci@0,0/pci8086,2779@1" unit-address = "0" interrupt-priorities = 5;
   name = "pciex14e4,1650" parent = "/pci@0,0/pci8086,2779@1" unit-address = "0,1" interrupt-priorities = 5;

6. After modifying the config either reboot the system or unplumb all
   interfaces and run the 'update_drv' command.

7. When the system has been reconfigured and the interfaces plumbed back up
   you can verify the new interrupt priority settings by running the
   following command as root:

   # echo "::interrupts -d" | mdb -k


IX. Tuning
    ======

There are a couple areas that can greatly effect network performance.

1. The number of Rx/Tx rings and MSIX interrupts.  See the Interrupt Management
   section above.  By default 4 rings are allocated per interface with one MSIX
   interrupt per ring.

2. Number of Rx/Tx buffer descriptors per ring.  By default bnxe allocates 1024
   descriptors and this value should be tweaked depending on the traffic
   characteristics, number of connections, etc.  We've tweaked this from 1024
   to 4096 with various results.  Just remember that the more connections, the
   slower the processing, and the higher chance of packets being dropped
   because of no buffers.

3. On Solaris 10 the GLDv3 layer does not support multiple rings for
   transmission.  Instead the fanning out of outgoing packets is done by the
   driver using a traditional hashing routine to route a packet on a Tx ring.
   See the 'route_tx_ring_policy' config in bnxe.conf.  The default is to hash
   on the TCP/UDP header information.

4. Number of IP Rx processing rings.  By default this is 2 IP rings per
   driver instance.  A good rule of thumb is to set this as a minimum of the
   number of driver rings.  Even better the number of CPUs on the system.
   Example, system >=16 CPUs:

     # ndd -get /dev/ip ip_soft_rings_cnt
     # ndd -set /dev/ip ip_soft_rings_cnt 16

   Or add this to /etc/system for persistence:

     set ip:ip_soft_rings_cnt=16

5. Lastly, make sure that hardware flow control is turned on.


X. Debugging
   =========

There are many statistics exposed via kstat by bnxe.  The main groups are:

  . 'intr' for interrupts stats
  . 'l2chip' for layer 2 chip stats
  . 'l2driver' for layer 2 driver stats
  . 'l2stats' for general layer 2 stats
  . 'link' for detailed link status
  . 'mac' for GLDv3 MAC layer stats
  . 'rxq#' for Rx ring stats
  . 'txq#' for Tx ring stats
  . 'fcoe' for FCoE stats
  . 'dcbx' for DCBX stats
  . 'stats' for general driver stats and version info

To get a list of all the individual statistics in these goups run:

  % kstat -m bnxe -i 0 -l

Additionally, a kmdb debugger module is installed with the BRCMbnxe package
that offers a much deeper level of debugging information in comparison to
kstat.  If the kernel is booted with the "-k" option the bnxe mdb module will
be automatically loaded if/when a panic occurs.  To load the module manually
when running "mdb -k" or "mdb -K" simply run the following mdb command:

  i386> ::load /kernel/kmdb/bnxe

  amd64> ::load /kernel/kmdb/amd64/bnxe

  sparc> ::load /kernel/kmdb/sparcv9/bnxe

This bnxe debugger module implements a single dcmd named "::bnxe":

  > ::help bnxe

  NAME
    bnxe - bnxe driver status

  SYNOPSIS
    [ addr ] ::bnxe [ -i # ] [ -a | -b | -r | -l | -c # | -f ]

  DESCRIPTION
    addr::  dev_info address ('::devbindings -q bnxe')
    -i #    driver instance # (instead of 'addr::')
    -a      show all (basic, interrupts, all chains, fcoe)
    -b      basic status (default)
    -r      interrupt status
    -l      mutex locks status
    -c #    status for chain # (-1 for all chains)
    -f      fcoe status

For example:

  # mdb -k
  > ::load /kernel/kmdb/amd64/bnxe
  > ::bnxe -i 8
  --------------------------------------------------

  bnxe8 -> ffffffff82adcbd0 (fffffe841cbce000)

  Version:            7.0.13     Firmware:              7.0.27.0
  ChipName:           BCM57712E  ChipID:                0x16630010
  SF/MF:              SF (1)

  L2 MacAddr:         00:10:18:6f:d4:48

  HwInitDone:         1          ClientsHw:             L2,FCoE
  ClientsDevi:        FCoE       ClientsBound:          L2,FCoE

  txMsgPullUp:        0

  memAllocBlocks:     21         memAllocDMAs:          299
  memAllocBARs:       5

  timerFired:         18605      timerReply:            18604
  timerNoReplyTotal:  0          timerNoReplyCurrent:   0
  timerDone:          0

  workQueueInstCnt:   0          workItemInstQueued:    161
  workItemInstError:  0          workItemInstComplete:  161

  workQueueDelayCnt:  0          workItemDelayQueued:   0
  workItemDelayError: 0          workItemDelayComplete: 0

  pcie_lane_width:    8          pcie_lane_speed:       5Gb
  intrAlloc:          6 MSIX     intrFired:             36625

  -- SB: Default --
     hwSpIdx:        18679        lcSpIdx:        18679
     hwAttnIdx:      10           lcAttnIdx:      10

  --------------------------------------------------
  >

There is a lot of info that the ::bnxe dcmd can display.  It is recommended
that all output for all interfaces (i.e. via "::bnxe -a") is gathered and filed
along with any bug reports.


XI. FCoE
    ====

    1. Features
       --------

    The following are the supported features of this driver (bnxef):

    * fcinfo(1M)
    * native multi-pathing
    * FC HBA API
    * FC-tapes
    * FC disks

    2. Features not supported
       ----------------------

    * NPIV feature not part of Solaris 10 and is only supported on Solaris 11.

    3. Supported FC/FCoE Devices
       -------------------------

    The bnxef QLogic 10Gb FCoE driver works with all the major FCoE/FC fabric
    devices. It is also compatible with all known FC disk devices and tape
    device working through the FCoE fabric.

    4. Starting and Stopping the FCoE Driver
       -------------------------------------

    By default, the bnxe driver starts FCoE (bnxef) on every interface that
    both supports and is licensed for FCoE. The FCoE port instantiated cannot
    be stopped or deleted when the system is running. 


    5. Configuring the FCoE Driver
       ---------------------------

    The bnxef driver can be configured via the bnxef.conf file installed under
    /kernel/drv.  When this config file is modified the system must be either
    rebooted or use the update_drv(1M) command to update the driver
    configuration.

    The details of the configurations parameters are detailed in the bnxef(7D)
    man page.  The default parameters should work for all conditions. There
    should be no need to update any parameters in this file.

    6. Debugging FCoE Driver
       ---------------------

    A kmdb debugger module for bnxef is installed with the BRCMbnxe package.
    If the kernel is booted with the "-k" option the bnxef mdb module will be
    automatically loaded if/when a panic occurs.  When the system is running
    and you would like to inspect some data structure pertaining to this
    driver, you can run from the command prompt as a super user the following
    commands.

      # mdb -k

    This will bring up the kernel debugger (without stopping the system)
    allowing you to inspect the kernel and or driver structures. The bnxef mdb
    module is installed at /kernel/kmdb/amd64/bnxef for 64-bit and
    /kernel/kmdb/bnxef for 32-bit. In order to load this module automatically
    one could create in the home directory for the 'root' user the dot file
    named ".mdbrc". This file should contain the following command.

      ::load /kernel/kmdb/amd64/bnxef

    For every problem identified, panic or otherwise, please invoke either "mdb
    -K" which will drop into kmdb or invoke "mdb -k" if the system is alive
    and it capable of running the above commands. Then issue the following
    command at mdb prompt.

      > ::bnxef_dump_all

    The above will dump a lot of data on terminal when invoked.  The data
    contains all the specific information about multiple instances, commands
    pending, and other details which could help debugging.

    In case of panic, collect the complete screen dump at the time of panic.
    Also collect the information dumped by the following two commands.

      > ::msgbuf

    This will dump the console messages from the message buffer.

    For other problems, having a network analyzer trace will greatly aid in
    debugging as well.

